System for knowledge discovery

ABSTRACT

An augmentation system configured to present global and local context sensitive augmented content in accordance with multiple criteria, including one or more user&#39;s preferences, and present the augmented content via a display layer on top of the original content. The augmentation system provides an efficient browsing, knowledge discovery and exploration experience. The original content remains visible to the user while the augmented content is being viewed and/or interacted with. The global context sensitive augmented content is generated using metrics based on the overall document or content being viewed, while the local context sensitive augmented content is generated using metrics based on the currently viewed or selected object within the document or content being viewed. Real-time and theme based augmentation are used to further enhance the user&#39;s experience. The local and global augmentation includes user customized augmented network of concepts and relationships that are of interest to knowledge discovery of the topic at hand.

PRIORITY CLAIM

This application claims priority to, and is a continuation of,application Ser. No. 13/573,564 filed on Sep. 24, 2012, entitled “Systemand method for augmented browsing and knowledge discovery”, which claimspriority to U.S. Provisional Patent Application entitled “System andMethod for Context Sensitive Augmented Learning and Browsing”, havingapplication No. 61/626,253, filed Sep. 23, 2011, and to U.S. ProvisionalPatent Application entitled “System and Method for Context SensitiveAugmented Learning and Browsing”, having application No. 61/743,047,filed Aug. 24, 2012. Each of the above named applications is herebyincorporated by reference herein in its entirety.

TECHNICAL FIELD

The subject of this application generally relates to informationretrieval and processing, knowledge discovery, and data presentationparticularly to systems and methods for improving data-exploration,learning and browsing and, more specifically, to context sensitive dataaugmentation for a richer user experience (UX) in browsing,data-exploration, knowledge discovery, and learning systems.

BACKGROUND

Having a context sensitive user interface which can automatically choosefrom a multiplicity of options based on the current or previous state(s)of a program operation can be found in current graphical user interface.For example: Clicking on a text document automatically opens thedocument in a word processing environment. The user does not have tospecify what type of program to use to open the file. Program files andtheir shortcuts (i.e. executable files) can be associated with certaintype of files, e.g. text document, and are automatically run by theoperating system when the user selects or double clicks the file.Similarly, the user-interface may also provide context sensitivefeedback, such as changing the appearance and/or color of the mousepointer or cursor. In addition, context sensitive feedback may also beused in video games where it change a button's function based on aplayer who is in a certain position or a place and needs to interactwith an object.

Relational databases are currently the predominant choice in storingdata like financial records, medical records, personal information,manufacturing and logistical data. Nowadays large-scale data orinformation processing can involve various types of collection,extraction, warehousing, analysis and statistics. For example,organizing and matching data by using some common characteristics foundwithin the data set would result in new groups of data that canorganized and are easier for many people to understand, search, indexand manipulate.

By describing the contents and context of data files, the quality ofprocessing the original data files can be greatly increased. Forexample, a webpage may include metadata specifying what language wasused in writing its code, what tools were used to create it, and whereto go for more on the subject, higher-level concepts that describe thedata. Thus allowing browsers to automatically improve the experience ofusers. The results of any large-scale data processing can be anextensive set of metadata, data, and relationships that may be used in asearch engine, for example, to provide a possible set of relatedinformation to a term that is used in a search query. For example,search engines have used and generated enormous amount of data andmetadata that is used to provide links to content that may be ofpossible interest to a user based on what the user is searching for.

As stored digital information has increased tremendously in size, theability of a user to use effectively personal data, corporate data, orpublically available data has also increased many folds although itstill falls short of the potential of reasoning about the large amountof data that is available and continues to grow at an astounding pace.Therefore, there exists a need to more effectively use and reason aboutthe data, and with more a richer augmented user experience whilereading, writing, searching, or using digital data information.

SUMMARY

This disclosure presents new and useful methods and systems to providemultilevel context sensitive augmented experience, browsing, dataexploration, knowledge discovery, and e-learning. In accordance with oneembodiment, this multilevel context sensitive augmented content ispresented using overlaid layers on top of the digital information(reference content or original content) being viewed by a user.Furthermore, the overlaid layers can be transparent or translucent for anon-obtrusive user experience. Thus, providing the user the ability tointeract with the original content while viewing a dynamically updatedaugmented content on top of the original content, the updated augmentedcontent is generated based at least on the user interaction with thereference content. Furthermore the user can manipulate the originalcontent and its associated or related categories and other relevantaugmentation data to generate more relevant and meaningful augmentationwhile viewing the augmented content on top of the reference content.

In accordance with one embodiment, a system for generating andpresenting augmented content on a translucent display layer overlaid ontop of a reference content display layer on the same display screen. Theaugmented content is generated using relevant features and filtersextracted from the reference content or the displayed portion of thereference content. A feature is a pattern that can be extracted orinferred from the content at hand. Feature extraction is the process ofreducing the dimensionality of a document by capturing a set of featureswhich reflect the most relevant and salient properties of that document.For example, a feature can be a keyword in the content, title of thecontent, or other metadata that can be extracted or inferred from thecontent, its link, or any embedded content or link to other content. Afeature could also correspond to a concept such as a name, a topic, oran event that can be extracted or inferred from the content. A group offeatures are combined together using an association rule to form apattern, a complex pattern, or a filter. A filter may comprise ordescribe a relationship between features, a collection of features, or agroup of features. A filter may also reflect a correlation between a setof features. A category is a grouping of features or a grouping ofmultiple sets of features. A category may correspond to a classificationof entities or concepts that share some property or relationships. Acategory maybe formed using a filter, group of filters, or anycombination of filters and features. An association rule to combine afeature, a set of features, a filter, or a set of filters can also beused to generate a category, a set of categories, a new feature, a newfilter, a new set of features, or a new set of filters. Thus, thegeneration of the augmented content can use (i) any one of a feature, afilter, a category, or (ii) any association in between, or acombination, of a filter, a feature, and a category. The generation ofaugmented content is further customized using user-relevantcharacteristics, attributes, history, and other relevant features inrelation to the reference content such as generic categories andrelationships. In addition, the user controls the position and size ofboth the reference content display layer and the augmented contentdisplay layers on the same display screen, as well as the ability of theuser to control the visibility and hiding of all display layers.Furthermore, the user controls the sharing of the same display screen bythe reference content and augmented content display layers.

In accordance with one embodiment, the system for generating andpresenting augmented content provides a set of augmentation filters:topics and categories based on the reference content to aid the user infurther customization of the augmentation filters to suite his/herinterests. The generated augmentation content is one or more of onlinedocuments, web pages, web links. The generated augmentation content canbe a customized version using a variety of ways such as presenting asummary of the augmented content, or in deleting un-necessary links andads. In accordance with one embodiment, the generated augmentationcontent is based at least on one of (i) a set of criteria associatedwith the reference content, (ii) user customization of augmentationfilters, (iii) user interaction with the reference content, and (iv)user interaction with the generated augmented content.

In accordance with one embodiment, the system can employ the samemethods and algorithms to enable the user to custom build a knowledgegraph of concepts and relationships based on information retrieved fromstructured and unstructured data residing in a private or public datastore or other public repositories. The Augmentation System relies onthese data sources along with the user's feedback and interests togenerate on the fly relevant augmentation data for the task at hand. Forexample, a physician can utilize this system to custom build a knowledgegraph for a patient based on the physician's experience and knowledge,the patient's history, the patient's known diseases, symptoms, andailments, and known public data related to the patient's case. Such asystem will enable the physician to make educated and informed decisionsinstead of being mired in a plethora of sources where it would beextremely hard for the physician to manually extract reliable andrelevant data in an efficient and useful way.

In accordance with one embodiment, the system for generating andpresenting augmented content dynamically updates the augmented contentby utilizing additional filters, metrics, and customization provided bythe user as a result of the generated augmented content. Furthermore,the user can save any or all the data associated with a particularsession of data augmentation. This will enable the user to build on theaugmentation of previous sessions.

In accordance with one embodiment, the system for generating andpresenting augmented content generates global and local augmentationcontent associated with the reference content and any selected orhighlighted part of it. For example, the system generates a plurality ofglobal augmentation content based on the augmentation filters associatedwith the overall reference content, and the system generates a pluralityof local augmentation content based on a specific part of the referencecontent that is selected or flagged by the user, or currently beingviewed by the user.

In accordance with one embodiment, the system for generating andpresenting augmented content enables collaborative augmentation, e.g. auser can share the generated augmented content with other users.Furthermore, the user can share the content augmentation filters or thesettings used to generate the augmented content with other users.

In accordance with one embodiment, the system for generating andpresenting augmented content enables a user to make use of nestedhierarchical content augmentation capabilities. A user can requestcontent augmentation using at least a portion of a previously generatedaugmented content. The previously generated augmented content serves asnew reference content for the system to generate and present to the usera new augmented content. The user can traverse the content augmentationgraph to further customize the content augmentation at any level.

In accordance with one embodiment, the display screen may be physicallyattached to an electronic device, e.g. a mobile device, a handhelddevice, a tablet, etc. . . . , or the display screen may physicallyseparate from the electronic device. For example a touch display where auser interacts with the display screen and controls both the positionand size of the various display layers on the display screen. Thedisplay screen can communicate with a remote electronic device such as aremote server, or a mobile device. Alternately, the user can control theposition and size of all display layers on the physically detacheddisplay screen using the electronic device.

In accordance with one embodiment, the user interaction with any one ofthe reference content, a portion of the reference content, an augmentedcontent, and a portion of an augmented content includes at least one ofa manipulation of a region of the first display layer, a manipulation ofa region of the second display layer, hiding of the first display layer,hiding of the second display layer, hiding a display layer, saving atleast a portion of the reference content, saving at least a portion ofthe augmented content, saving at least a portion of one or more of a setof filters, a set of features, a set of categories, a set of metrics, aset of user preferences, modifying the translucency of a display layer,a selection of displayed content using a region of the first displayscreen, a manipulation of displayed content using a region of the firstdisplay screen, one or more user gesture made onto the first displayscreen, an activation of a button of the first display screen, a userinput made using the electronic device, and using a human interfacedevice to communicate the user interaction to the electronic device.

In accordance with one embodiment, the reference content, local content,augmentation content are displayed using multiple display layers bymeans of one or more display screens. The display screen compriseselectronic system to receive and/or transmit information to anelectronic device. The user interaction with the reference contentincludes the manipulation of one or more regions of at least one displaylayer, a manipulation of one or more regions of at least one displaylayer of the augmented content, hiding of any one or more of the displaylayers, saving the first set of augmented content, saving a portion ofthe first set of augmented content, modifying the translucency of anyone of the display layers, a selection or a manipulation of one or moreregions of any one of the display screens, one or more user gesture madeonto the display screen, an activation of a button of the displayscreen, an activation of a button of the electronic device, and using ahuman interface device to communicate the user interaction to theelectronic device or to the display screen.

In accordance with one embodiment, this disclosure refers to augmentinga given content based on a number of manually defined and automaticallyextracted parameters to generate a set of local and global dataelements. The set of local and global data elements can be used in avariety of application specific augmentation systems to enhance a user'sexperience while interacting with the given content.

In accordance with one embodiment, this disclosure facilitates theconstruction and presentation of a user-customized network of concepts,objects and relationships that serve to augment the content at hand forthe purpose of knowledge discovery, learning, and a richer userexperience in browsing and/or interacting with data information.Furthermore, the constructed network can be saved and further augmentedover time for richer and more efficient user experience. This is incontrast to having a pre-built network of concepts and relationshipsthat a user can access. This system generates a network that can becustomized and tailored based on the user's interests.

In accordance with one embodiment, this disclosure facilitates a systemthat provides the user the ability to fully control the generatedaugmented content by virtue of changing the scope of certain topics,e.g. expanding or specifying a narrower sub-topic, based at least on oneof a defined theme, predefined themes, and categories. Therefore, theaugmented content can serve to further explain, define, and to elaborateand expound on reference content or a selected portion of referencecontent being viewed, observed, or interacted with by a user.

In accordance with one embodiment, this disclosure can be used toaggregate information related to a reference or selected content bycustomizing the augmentation filters to achieve the desired or intendedresults. For example, the information, reference content, or thegenerated augmented content can include rich media like video, audio,images as well as text. Various filters can be customized by the user toenable a user to increase the relevance of the generated augmentedcontent to the intended user objective. In addition, a hierarchicalsystem of content augmentation maybe defined and customized by aselected theme or a category. The generated augmented content and itsdisplay layers can be monetized for ads and other monetization purposes.

In accordance with one embodiment, this disclosure enables real-timemanipulation of reference and augmented content for enhanced and richerUser Experience (UX). In addition, collaboration and sharing ofaugmented content provides an increase in value and productivity to auser. Similarly, collaboration and sharing of augmentations filters andsettings provide additional richness and ease of viewing, browsing,sharing, and manipulation of reference and augmented content.Furthermore, the user is able to control the presentation style of thegenerated augmented content, e.g. as raw links, concise summary ofaugmented content, or other methods that capture the essence of theaugmented content. The presentation style of the generated augmentedcontent maybe for data analysis, research, information, monetization,commercial, or educational purposes.

BRIEF DESCRIPTION OF THE DRAWINGS

The attached drawings, which are incorporated into and constitute a partof this specification, illustrate one or more examples of embodiments.These drawings together with the description of example embodimentsserve to explain the principles and implementations of the embodiments.

FIG. 1 is a block diagram of a data augmentation system 100 that is usedto generate augmented content for a given reference content.

FIG. 2 is a block diagram of a hierarchical augmentation system 200 thatis used to support the generation of multilevel augmented content usingmultiple reference content.

FIG. 3 is a block diagram of a hierarchical augmentation system 300 thatis used to support the generation of multilevel augmented content usingmultiple reference content along with a controller that manages thenested augmentation functions and the user's interaction with thegenerated augmented content.

FIG. 4 is a block diagram of a data augmentation system 400 having thecapability of manipulating, controlling and displaying both thereference content and the augmented content simultaneously anddynamically.

FIG. 5 is a block diagram of a relevant augmented content extraction 500that is used as a subsystem of a data augmentation system.

FIG. 6 is a block diagram of a relevant augmented content extraction 600that is used as a subsystem of a data augmentation system.

FIG. 7 is a display example of the generated augmented content of usingmultiple display layers.

FIG. 8 is a display example of a simulated use case of a dataaugmentation system invoked while viewing a news article.

FIG. 9 is a display example of a simulated use case of a userinteraction with a data augmentation system invoked while viewing a newsarticle.

DETAILED DESCRIPTION

The present disclosure presents techniques, systems and methods toprovide a user with global and local context sensitive augmented contentto enhance the user experience while interacting with digitalinformation be it while reading, writing, drawing, browsing, searching,viewing, or using digital data information such as financial, medical,business or corporate data, social media data, or any data that isaccessible locally, on the web and/or remotely through a network or webbased services. These techniques, systems and methods are applicable tovarious computing platforms such as hand-held devices, desktopcomputers, notebook computers, mobile devices, as well as computeservers.

The term “coupled” is defined as connected, although not necessarilydirectly, and not necessarily mechanically. The terms “a” and “an” aredefined as one or more unless this disclosure explicitly requiresotherwise. The terms “comprise” (and any form of comprise, such as“comprises” and “comprising”), “have” (and any form of have, such as“has” and “having”), “include” (and any form of include, such as“includes” and “including”) and “contain” (and any form of contain, suchas “contains” and “containing”) are open-ended linking verbs. As aresult, a method or device that “comprises,” “has,” “includes” or“contains” one or more steps or elements possesses those one or moresteps or elements, but is not limited to possessing only those one ormore elements. Likewise, a step of a method or an element of a devicethat “comprises,” “has,” “includes” or “contains” one or more featurespossesses those one or more features, but is not limited to possessingonly those one or more features. Furthermore, a device or structure thatis configured in a certain way is configured in at least that way, butmay also be configured in ways that are not listed.

Example embodiments are described herein in the context of a system ofone or more mobile device, electronic device, handheld device,computers, servers, firmware, and software. Those of ordinary skill inthe art will realize that the following description is illustrative onlyand is not intended to be in any way limiting. Other embodiments willreadily suggest themselves to such skilled persons having the benefit ofthis disclosure. Reference will now be made in detail to implementationsof the example embodiments as illustrated in the accompanying drawings.

In the interest of clarity, not all of the routine features of theimplementations described herein are shown and described. It will, ofcourse, be appreciated that in the development of any such actualimplementation, numerous implementation-specific andapplication-specific decisions must be made in order to achieve thedeveloper's specific goals, such as compliance with business-relatedconstraints, and that these specific goals will vary from oneimplementation to another and from one developer to another. Moreover,it will be appreciated that such a development effort might be complexand time-consuming, but would nevertheless be a routine undertaking ofengineering for those of ordinary skill in the art having the benefit ofthis disclosure.

In accordance with this disclosure, the components, process steps,and/or data structures described herein may be implemented using varioustypes of operating systems, software development platforms, computingplatforms, computer programs, and/or general purpose machines. Inaddition, those of ordinary skill in the art will recognize that devicesof a less general purpose nature, or having limited resources mayrequire modification of an implementation of an illustrated embodimentwhich maybe done without departing from the scope and spirit of theinventive concepts disclosed herein. Where a method comprising a seriesof process steps is implemented by a computer or a machine and thoseprocess steps can be stored as a series of instructions readable by themachine, they may be stored on a tangible medium such as a computermemory device (e.g., ROM (Read Only Memory), PROM (Programmable ReadOnly Memory), EEPROM (Electrically Erasable Programmable Read OnlyMemory), FLASH Memory, Jump Drive, and the like), magnetic storagemedium (e.g., tape, magnetic disk drive, and the like), optical storagemedium (e.g., CD-ROM, DVD-ROM, paper card, paper tape and the like) andother types of memory.

The term “exemplary” is used exclusively herein to mean “serving as anexample, instance or illustration.” Any embodiment described herein as“exemplary” is not necessarily to be construed as preferred oradvantageous over other embodiments.

In general, various information processing techniques and algorithms canbe used to provide the augmented data system with global and localcontext sensitive augmented content. In the following paragraphs certaindefinitions and representations of data flow models are presented anddiscussed without limitations on how each model may be implementedwhether by hardware, software, firmware, or any combination thereof.

Multilevel, e.g. global and local, context sensitive augmented contentwould increase productivity and enhance a user's experience whileviewing or interacting with data for the purpose of learning, reading,writing, drawing, browsing, searching, discovering, viewing images orany type of user interaction with digital data information whetherstructured or unstructured (e.g. financial, health, manufacturing, andcorporate data). The digital data information may be stored locally orremotely via a corporate server or in the cloud. Additionally private aswell as public sources of data may be used or selected by the user forthe ultimate personalized range of choices that may be used to furthernarrow down or expand the augmented content being presented.

Additionally, multilevel context sensitive augmented content wouldincrease productivity and enhance business intelligence for theenterprise by providing context sensitive augmented content that isgenerated by dynamically mining and analyzing structured andunstructured enterprise data and/or possibly leveraging structured andunstructured publically available data for further improving userexperience. In addition, multilevel context sensitive contentaugmentation filters provide the ability to dynamically mine data on thefly based on modification of a new input from a user. For example, a newinput from a user can be the selection of a new text or a portion of thereference content, or it can be a feedback provided such as elevatingthe priority or weight (e.g. like) or decreasing the priority or weight(e.g. dislike, delete, dismiss) of a single augmented content, acategory of augmented content, or a theme of augmented content.Furthermore, leveraging the history and/or user personal preferences,the multilevel context sensitive augmented content can be further intune with what the user would like to see or expects to see in theaugmented content being generated and presented.

In accordance with one embodiment, a feature of the multilevel contextsensitive augmented content is that the augmented content is generatedeither in the cloud or locally using sophisticated information retrievalalgorithms or using a set of heuristics so as to enable large-scale dataprocessing, information retrieval, and web mining. Knowing thatextracting a feature set from a web page is a problem that is known andvarious algorithms, methods, and research into various solutions havebeen made, this system can use existing research or methodologies toextract a feature set. Furthermore, this system employs a set ofheuristics and metrics that efficiently extract a set of features thatcharacterize the reference content at hand. These heuristics rely onembedded hints, metrics, metadata, or other embedded knowledge andinformation that can be extracted from the structure, url link, embeddedlinks, title of the document, or other types of data that may bedirectly or indirectly related to the reference content along withfeedback provided by the user.

In accordance with one embodiment, a feature of the multilevel contextsensitive augmented content is that the information retrieved andknowledge constructed can be saved and called upon in futureaugmentation tasks and sessions.

In accordance with one embodiment, a feature of the multilevel contextsensitive augmented content is that the augmented content is presentedthrough a translucent layer on top of the original content being viewedby the user. Hence a non-obtrusive content augmentation that is hiddenor made available whenever a user disables or enables the global andlocal context sensitive augmented content application. Relevantaugmented content are displayed on top of a translucent layer on top ofthe original content being viewed by the user. Hence, the augmentationsystem provides a less obtrusive and more efficient interaction,browsing and exploration experience.

In accordance with one embodiment, a multilevel corresponds to at leasttwo levels, a global level and a local level. A global and a localrelevant features of reference content, maybe defined as a globalrelevant feature corresponding to a feature or a theme common throughoutthe reference content, and a local relevant feature corresponding to afeature strongly related to a locality within the reference content. Onemethod of dynamically updating augmented content can be achieved byleveraging real-time user feedback, such as elevating priority ordismissing augmented content as being presented to the user. If anaugmented content's priority is elevated, its weight increases as wellas the metadata that describes this augmented content gets promotedwhich in turn updates existing augmentation filters as well asgenerating and presenting new augmentation content based on the newmetrics. For example, if an augmented content describing certain publicpolicy information is promoted, then that augmented content's priorityis increased, and the priorities of all augmented content that referencesome public policy, or government policy get increased. In addition, theaugmented content can be dynamically updated based on user interaction,e.g. selection and/or clicking, within the reference or augmentedcontent in real time. There are various means to implement the augmentedcontent presentation layers such as dials for global and local augmentedcontent, or a scroll-area of small windows for various augmentedcontent. Describing all these various means to implement the augmentedcontent presentation layer is not necessary to understand thisdisclosure. Furthermore, a person skilled in the art would understandand would be able to employ many different means to implement augmentedcontent presentation layers without departing from the spirit of thisdisclosure.

In accordance with one embodiment, while generating augmented contentmay result in a lot of data that cannot be shown on the display, thisdata can be stored in a deep queue. A deep queue means that there ismore augmented content (data) in the queue than what is displayed on thescreen. For example, not all mined augmented content can be displayedsimultaneously due to physical screen size limitations or the displaylayer size. A user can hover over the queue or press an arrow to scrollthrough the augmented content in the queue. In addition, it is importantto note that the augmented content being presented to the user maycomprise actual data, snap shot of the actual data, a processed portionof the actual data, or a link to the location where the actual data canbe retrieved.

Theme-based augmented content can further enhance a user's experience bypresenting a set of themes. In accordance with one embodiment, when theuser selects or deselects a theme, a new or updated augmented content ispresented to the user. An option to expedite augmentation and improvethe quality is to rely on the user's preferences and feedback. When theapplication is invoked, a set of categories/themes can be presented tothe user. These constitute metadata. By relying on the user choices ofthemes, augmentation can be enhanced and filtered. For example, aresearch paper that deals with AIDS virus would trigger a set of themessuch as Pharmaceuticals; Discrimination, etc. . . . The user who isinterested in science and pharmacology but not in the social aspectsrelated to AIDS would deselect ‘Discrimination’. Thus, all augmentedcontent presented will be tailored to refer to categories that arerelated to science and other related aspects of the research. The themecan further be defined by a category or a set of related categories.This will serve to prune the augmented data and only present therelevant data that is of interest to the user and the task he iscarrying out at that moment.

Multilevel context sensitive augmented content application can beimplemented as a stand-alone application, on top of another application,or as an extension for applications, e.g. a browser extension. Inaccordance with one embodiment, further refinement or fine tuning ofvarious options for customization of augmentation system such asaggregating, mining, filtering, and presenting various aspect of data ormetadata can be performed dynamically in real-time. In addition, thecustomization of augmentation system maybe performed based on at leastone or more of a user's feedback, attributes, characteristics, theme,topics, and interests. Also when augmentation system presents a list oftags/categories, the user can provide feedback in the form orliking/disliking the tag. This is similar to promoting or dismissing anaugmented content. Therefore, in accordance with one embodiment, theaugmented content can be updated live. Furthermore, this user's feedbackwould also result in updating various subsystems such as the underlyingdata-mining, statistical computing algorithms, or machine-learningalgorithms or other information retrieval algorithms or heuristics.These updated subsystems are used to generate or create new signatures,metrics, or features which are based on user's feedback, e.g.liked/disliked tags, where the new signatures are used to generate newaugmented content or update the currently presented augmented content.

In accordance with one embodiment, a feature of a system for generatingand presenting multilevel context sensitive augmented content is theability to utilize online and offline mining and analytics foraugmentation. For example, mining and processing in real-time or inbatch mode and store data in a data store (local or remote) orpresenting real-time augmented content to the user. The stored data canbe used for future augmentation. Metadata and other relevant dataelements can also be annotated in real-time to capture user'spreferences and experiences. In addition, metadata and other relevantdata elements can be stored in a central repository to be leveraged forfuture augmentation of same or similar content. A brief description ofmetadata is that it is data that describes other data. For example:‘public health’ is a category that encompasses diseases. This higherlevel category ‘public health’ is a metadata for diseases.

In accordance with one embodiment, a multilevel context sensitiveaugmented content system uses at least two levels, a global level and alocal level. The following explains the difference between global andlocal augmented content. Global augmented content refers to augmenteddata that pertain to the overall document that the user is currentlybrowsing, exploring, or interacting with. A local augmented content canrefer to augmented content based on a particular piece, paragraph,sentence, word, image, icon, symbol, etc. . . . of that document thatthe user is currently browsing, exploring, or interacting with. Globaland local augmented content are presented using a dynamic deep queue,and the user can control the displaying of at least a portion of theaugmented content. Content sources for augmentation can be provided frommany sources. An example of such content sources includes but is notlimited to a user's own documents and data on desktop, web-content,social media sites, enterprise data-marts, and local and remote datastores, ontologies, other categorization, and/or semantic orrelationship graphs.

The multilevel context sensitive augmented content can be successfullyimplemented to augment a user's browsing experience as discussed above.In accordance with one embodiment, a system for generating andpresenting multilevel context sensitive augmented content can besuccessfully implemented as an application for augmented user experience(UX). The system can increase productivity, provides augmenteddata-mining and data-exploration platform, augmented e-learning ande-research system, augmented desktop-based and mobile-based browsing,exploration, research, discovery, and learning platforms, dataaugmentation for better healthcare products and services, dataaugmentation for better educational products and services, augmentationsystem for better content management and relationship platform for bothenterprise and consumer applications, enhanced online-shopping researchand UX, enhanced marketing campaigns, an enhanced news access UX are butto name a few of application benefiting from a system for generating andpresenting multilevel context sensitive augmented content.

Semantic processing is the process of reasoning about the underlyingconcepts and expressing their relationships. In addition to variousaugmentation methods as described above, the following semantic basedtechniques can also be used in a system for generating and presentingaugmented content. In accordance with one embodiment, utilizing existingtags in public sources, utilizing batch-processed tags as a cloudapplication, semantic processing of selected content to generate a matchto an existing tag, semantic processing to generate augmented content onthe fly and utilizing user's feedback for promoting and dismissingaugmented content are but examples for methods to provide a betteruser-relevant augmented content. Generating augmented content on the flycan also be accomplished by using a feedback mechanism provided by theuser to enable mining and generating of new augmented data to bepresented to the user.

In accordance with one embodiment, a system for generating andpresenting multilevel context sensitive augmented content is used toimprove the analytics of large data sets by leveraging pre-processeddata and already generated relationships. Given a content that is theresult of a statistical data mining and exploration functions on a smallor large amounts data—be it remote or local—extract the correlationmetrics and other signatures that demonstrate a meta-relationship andleverage it in other data-mining, analytics, and to generate augmentedcontent. For example: When a user presents some key words to a searchengine, the user gets a set of links that are related in addition tosome ads that could very well be related to the key words you haveentered or to some personal data known or extracted of the user. Thesepresented links and ads have gone through a huge amount of processingand computation in the cloud. By knowing that a relationship or ameta-relationship exists between the keywords, links, and may be othercontent pushed to the user like ads, the analytics operation canleverage them and extract, store, and leverage these signatures forfuture browsing or for presenting context sensitive augmented content.

In accordance with one embodiment, the content presented to the searchengine can be either parsed from the html or other format or interfaceproduced by a data provider. Or, it can be scanned through OCR if thedata format is encrypted. This ability to take a snap shot of a screenand analyzes and leverages its data and relationships empowers andsimplifies the augmentation and analytics processes and improves thethroughput since the signatures/correlation metrics extracted are aresult of processing a significantly smaller set of data. Therefore, theperformance gain of a system for generating and presenting multilevelcontext sensitive augmented content is orders of magnitude compared tomining massive data sets in the cloud.

In accordance with one embodiment, a system for generating andpresenting multilevel context sensitive augmented content presents theaugmented content along with the reference content using two or moredifferent presentation layers displayed using the same display screen.In addition, the system provides the ability to customize the generationof augmented data in situ (in place) while working on original orreference content, where the augmented data can be displayed on see thrupresentation layers so as not to obscure the original or referencecontent and to maximize use of the display screen, and/or the displayingarea.

In accordance with one embodiment, a system for generating andpresenting multilevel context sensitive augmented content utilizesdynamic updates of displayed augmented content using presentation layerswhile a user views and manipulates reference content displayed usinganother presentation layer. It is preferable to use a translucentpresentation layer for the augmented content presentation layer that islocated on top of the displayed reference content so that the user caneasily manipulate or interact with the reference content whilesimultaneously viewing the dynamically updated augmented content. As canbe easily appreciated by person skilled in the art that displayingrelevant augmented data in a separate tab or page would result in lossof context relationship and provides a less efficient and less friendlyuser experience. Similarly, displaying the augmented content on thesidebars is possible as well. However it consumes screen space andclutters displaying of the reference content. Therefore, the ability tokeep the reference content accessible to the user while displaying theaugmented data on top of the original content provides a much smootherand efficient user experience. Furthermore, the user can easily hide,size, move, or display the augmented content without affecting thereference content.

In accordance with one embodiment, a system for generating andpresenting multilevel context sensitive augmented content enables a userthe ability to associate any of the augmented content with the referencecontent or an attribute of the reference content source using one ormore types of metadata. The system enables the user to save theassociated metadata for future use or sessions. For example, theassociation of metadata can be accomplished by embedding a link in thetext, by associating a link with a text, or by associating any data ormetadata with the reference content or any part of the referencecontent. Moreover, the user has the ability to specify a category ormore as a source or criterion of augmentation. The user can also defineassociation rules that join a group of attributes, categories, and othermetrics together to provide a richer input to aid the augmentationsystem to generate more relevant augmentation content. For example, anenterprise sales projection document can always be augmented with anydata source or data documents that generated the projection. Thecriterion is a category that says source sales data and not necessarilythe exact data documents. The sales data can be extracted automaticallyby the augmentation system. Utilizing selected or provided categories ofinterest, the augmentation system can carry out an updating procedurefor any associated data or metadata for any other reference content.Furthermore, the augmented content is displayed using translucent layersso that the user always sees and has access to the original or referencecontent. The user is able to access, browse, move, select, hide, tap,scroll, or interact with the reference or augmented content while thesystem dynamically generates and displays an updated augmented contentusing the augmentation presentation layer. It is noted that the userinteraction with the reference or augmented content can result in havinga new reference content that the user wishes to interact with, hence, anew augmented content is generated and displayed. The system keeps trackof and saves certain information regarding this nested augmentationlevel. The system provides the user the ability to switch back and forthbetween various nested augmentation levels as well as saving or sharingthe augmentation filters or settings used for a particular session.

In accordance with one embodiment, further enhancement of the userexperience is achieved by enabling the user to change the skin (or lookof a user interface UI) of the augmentation system. For example, thesame components of a UI (buttons, options, data) can be displayed on thescreen in a variety of ways. Usually, a library of templates and coloroptions can be provided to allow the user to customize the display ofthe augmented content presented by the application. In addition, theglobal augmented content and local augmented content can be displayedusing one or more different regions of the screen, or displaying theglobal and local links to the augmented content in two concentriccircles around the reference content. The enhancement of the userexperience is achieved by enabling the user to choose the most efficientway to utilize and display the augmented content.

In accordance with one embodiment, user selectable skins can also beused to cover or hide pushed content that may exist or embedded in thereference content being viewed. User selectable areas of a skin can beused to enable the display of user selected content such as images oraugmented content, or pushed content such as advertisement. For example,an ad for tickets to a local concert when the user is browsing aspecific artist, or an ad for a book that relates to a global or localaugmented content of the user reference or currently viewed augmentedcontent, or any other monetization mechanism based on the augmentationprocess. The enhancement of the user experience includes a nestedmultilevel context sensitive augmented content where the augmentedcontent presented to the user can be further enhanced as a function ofthe various nested levels. The augmented content is presented whilekeeping track of the current content being viewed in relationship to theoriginal content that the user started with and all levels in between.This provides a hierarchical augmentation system that enables the userto access and build nested levels of augmentation.

In accordance with one embodiment, the user interface, or UI, for asystem for generating and presenting multilevel context sensitiveaugmented content can be launched or started automatically, and canreside in the background and stays hidden from view until the userinvokes a predefined programming function to enable the UIfunctionality. For example, a single tap, hot-key, function-key, agesture, or a multiple or a combination of actions acted upon a contentwould cause the transparent augmentation layer to be shown with theaugmented content and in accordance with user preferences, such as tags,skins, themes, etc. . . . Selecting content presents or updates theaugmented content already presented. Visiting an augmentation linkresults in completely or partially (split screen) covering the referencecontent or original layer comprising the original content. The UIprovides the user the ability to navigate nested augmented content orjump back to reference or original content.

In accordance with one embodiment, additional system and UI features canfurther be used to increase the overall efficiency and provide a betteruser experience. For example, saving the augmented content metrics inuser history, and using history to enhance and/or tailor analytics andaugmentation as would be more relevant to each individual user or groupof users such as in corporate environment. Metrics here refer to thegenerated signatures as mentioned above. Also, it refers to anyannotations that are provided by the user such as priority,liking/promoting an augmented content or dismissing it. This can bestored for future sessions as well as using the augmented contentpromotion and dismissal to enhance augmentation in real time. Usingskins that cover an undesirable part of the screen, e.g. side columnswhere ads are pushed. The skin may be used for further customization ofthe viewed screen and potentially could be monetized and leveraged topresent relevant augmented content that is paid for by the user, such asads for objects, e.g. books, related to the content of a referencearticle.

In accordance with one embodiment, a system for generating andpresenting multilevel context sensitive augmented content providesdynamic user-guided and customized context-sensitive data augmentationto facilitate learning, exploration and knowledge discovery. The systemprovides simultaneous interaction with the augmentation layer and thecontent layer. The system generates augmentation data based onuser-defined metrics and filters such as themes, categories of interest,document content and/or part of it. The generated data is not a rigidaugmented content. The generated augmented content is any data, concept,and relationships that are presented as a result of the data mining andprocessing of the original content and the user-defined metrics andfilters.

In accordance with one embodiment, the system utilizes dynamic andinteractive methods to successively refine and tailor the augmentedcontent based on a user's guidelines, filters, and metrics. The systemrelies on a variety of sources for content augmentation by accessing anyonline or offline databases, crowd-sourced databases, or open databases.Furthermore, over time, a custom built graph of concepts andrelationships can be built between different pieces of data as they areprocessed and augmented based on the user's filters and metrics toimprove the performance of the system and the User Experience. Thesystem provides a context-sensitive hierarchical augmentation frameworkfor deeper and expansive exploration and knowledge discovery. The systemenables construction of a customized graph of data, concepts, andrelationships based on the filters and metrics provided even in theabsence of content. Content can be generated on the fly for furtherexploration.

In accordance with one embodiment, the system enables sharing ofaugmented data and the associated metrics that generated them. Thisenables richer knowledge discovery by further refining a user'saugmented data based on other users' augmented content. This is usefulfor collaborative research and knowledge discovery. The system can belaunched from offline and online documents or reference content togenerate the augmentation content, data and graph of relationshipsamongst the concepts represented by the augmented content.

In accordance with one embodiment, the system provides a UI to displayand manipulate reference content and augmented content concurrently,dynamically, and interactively. The system provides one ore moretranslucent layers on top of the reference content to show the augmentedcontent. Translucent layers facilitate displaying the reference contentas well as the augmented content. Translucent layers can fully orpartially cover the original content. Augmentation layers can be hidden,minimized ((shown as icon), or moved around on the display screen tofacilitate easier display and interaction with the reference content.The system enables the user to manipulate and control a set of displaylayers (reference content layer, and/or augmentation display layers) ina very flexible fashion such that the user can size up, down, move,show, hide any of those display layers. The system provides anintuitive, rich, and friendly UX for data exploration and knowledgediscovery on small and large display screens. In particular, displayingof the augmented content concurrently and interactively on the originalcontent empowers the user to use this system on smart phones, tablets,and any other display. Furthermore, the system provides means to insertadditional content on the augmentation layers based on analytics on theaugmented content and the original content.

A block diagram of an Augmentation System 100 is shown in FIG. 1. AReference Content 105 corresponds to any electronic document or web pagethat a user wants to invoke the Augmentation System 100 to get AugmentedContent 190. The Reference Content 105 can be stored locally in a memorysubsystem of an electronic device, a memory subsystem of a displayscreen device, or is accessed from a remote location via a wired orwireless communication system. The communication system could use theinternet, a cloud, a data store, a computing device, server or adatabase via a wired or wireless networking link. The augmented contentis a generated content by the Augmentation System 100 based on theReference Content 105 using a set of features, filters, and categorieswhich are produced by at least one of an Extract Features 120, ExtractCategories 125, and Update Categories 137 subsystems as shown in FIG. 1.

A Local Content 110 is a selected portion of the Reference Content 105which the user wishes to get more specific augmentation about, or thatis a portion of the Reference Content 105 that the user is interactingwith. Furthermore, the Local Content 110 may also be automaticallyselected, tagged, managed, or generated by the Augmentation System 100,e.g. based on a displayed portion of the Reference Content 105 or a userinteraction with a portion of the Reference Content 105. Furthermore,the presentation and/or the displaying of the Augmented Content 190 ismanaged using Manage RAC 145 (RAC refers to Relevant Augmented Content)to control a Display Queue 165 and Display RAC 170.

The Augmentation System 100 generates Augmented Content 190 byfacilitating the construction of a user-customized network of concepts,objects and relationships that serve to augment the Reference Content105 at hand for the purpose of knowledge discovery, learning, and aricher user experience in browsing and/or interacting with datainformation. This Augmentation System 100 generates any one of a networkof concepts, a network of objects, and a network of relationships usingone or more of a set of features, a set of filters, and a set ofcategories. Each of the set of features, the set of filters, and the setof categories can be customized and tailored based on the user'sinterests and input. The constructed network can be saved and furtheraugmented over time for richer and more efficient user experience.

The Extract Features 120 subsystem extracts a set of features from theReference Content 105. General Features 117 can provide a set offeatures that can be updated and tailored overtime to at least one of aspecific user, specific project, specific objective, and specificsubject. Extract Features 120 generates a set of filters that denotesthe desired concepts for augmentation. For example, these concepts couldbe names of people, history, events, topics, or other metadata. Thesedata are either computed on the fly or pre-computed and stored locallyor remotely for current or subsequent augmentation sessions. Thisextraction process is based on embedded data in at least one of theReference Content 105, in a linked content to the Reference Content 105,metadata of the Reference Content 105, e.g. a title of the ReferenceContent 105, linked content to the Local Content 110, and semanticinformation that are either associated with the Reference Content 105 orthat can be extracted/aggregated from the Reference Content 105. Otherdata that can be extracted or inferred can be further used forconstructing a more meaningful feature set by utilizing a variety ofinformation retrieval, extraction, and inference algorithms and methods.There is large body of work on feature extraction that utilizes thecloud as well as other large-scale solutions. These approaches can beleveraged by the Extract Features 120 along with flexible and efficientalgorithms to generate a set of features on the fly based on the metricsand signatures mentioned earlier. Furthermore, any part of theAugmentation System 100 can be run remotely on a server or in theclouds, or it can be run locally on the host device.

The Extract Categories 125 function uses a set of categories or topicsthat are extracted based on the data that can be associated or extractedfrom the Reference Content 105. This data can be either metadata or anyother related data to the Reference Content 105. The Extract Categories125 extracts a set of categories from the Reference Content 105 and itsassociated links and data. Also, the system utilizes any embeddedcategories or metadata that are either embedded in the link or attachedto the Reference Content 105. The extracted categories can also describemetadata about the topic at hand. For example, if the reference contentis an article about AIDS, there are many categories that can augmentdata about AIDS. For example, a set of categories can be: History ofAIDS, Science of AIDS, Social Impact of AIDS, Symptoms of AIDS, etc. . .. A user may only be interested in the science of AIDS, so a user willinteract with the presented categories, e.g. by deselecting allcategories that are not related to science, and this will impact the setof features that are used in augmenting the Reference Content 105. Otherdata that can be extracted or inferred can be further used forconstructing a more meaningful category set by utilizing a variety ofinformation retrieval, extraction, and inference algorithms and methods.In addition, a General Categories 115, as shown in FIG. 1, is a set ofdefault categories that the Update Categories 137 processes to reflectthe user's interests. For example, the General Categories 115 can beBusiness, Politics, Education, Research, Health, Technology, etc. . . .The Update Categories 137 may use this optional input from the user tobias the augmentation to the categories of interest. This optional inputcan be stored and updated over time.

The interaction of a user with the Augmented Content 190 maybeaccomplished in a variety of ways. For example, the user may select oneor more of the presented categories for removal, selection, decreasingpriority, and increasing priority. The user may also define, modify, orinteract with an association rule to aid Extract Features 120 togenerate a more useful set of filters for better augmented content. Theassociation rule can leverage, use, or join one or more categories,features, filters, or concepts to (i) generate a new set of features,filters, categories, or Augmented Content 190, and (ii) to modify one ormore of the set of features, filters, or categories which are being usedto generate the Augmented Content 190. Based on the General Categories115 and user's interaction, further categorization and featureextraction will be biased towards the user's interaction or input. Thisis an optional input that is used to customize the Augmented Content 190based on a user's needs, the user's interaction with Augmented Content190, or to aid the Augmentation System 100 to provide more relevantAugmented Content 190 for a specific purpose. Upon a user's interactionwith the Augmented Content 190, an Update 130 function enables theuser's input to be considered by Update Categories 137, e.g. a user maychoose to delete some of the default/general categories that are not ofinterest or to elevate the priorities of some of those categories. Whendeleting categories, the Update Categories 137 will reduce the weight ofthe features that are related to those categories. When categories areelevated in priority, the Update Categories 137 increases the weightgiven to those features that are related to those categories. Thus,affecting and updating the Augmented Content 190 presented to the user.

An Update Filters 150 is used to indicate a user's preference for afeature or automatic feedback based on user's interaction with theAugmented Content 190. For example, when one or more of the ReferenceContent 105, Local Content 110, and Augmented Content 190 get updated orinteracted with by a user, then more clues and feedback can be gatheredfrom the updated list or the user's interaction as to revise thefeatures and categories that are of interest to the user in real time.However, the user may choose not to update the features and categories,and the Augmentation System 100 provides the user the ability to controlhow and when the Augmented Content 190 is generated and/or updated.

An Update Features & Categories 135 subsystem receives a first set offeatures from the Extract Features 120 subsystem, a first set ofcategories from the Extract Categories 125, and/or an updated set ofcategories from the Update Categories 137, and/or an Update Filters 150.Update Features & Categories 135 manages and controls the updating ofthe actual features and categories sets including any decision makingbased on the user input or interaction. The Update Features & Categories135 may communicate with any one of Extract Features 120, ExtractCategories 125, and Update Categories 137 to generate more features andcategories based on a variety of parameters including the user'spreferences. Furthermore, Update Features & Categories 135 also handlesupdating relationships and cleaning up for those features and categoriesthat were updated by the user.

A Compile RAC 140 subsystem receives a set categories and a set featuresfrom the Update Features & Categories 135 subsystem. Compile RAC 140includes a variety of functions and algorithms such as machine-learning,data mining and extraction, web crawling, data-mart accessing,extraction and processing functions, and other intelligent algorithmsand approaches are used to compile a set of relevant augmented contentor pages (RACs) based on at least one of the Reference Content 105,Local Content 110, and the interest of the user. Managed RAC 145subsystem is the controller that manages the presentation of theAugmented Content 190 via a Display Queue 165 and Display RAC 170. TheAugmentation System 100 listens to inputs from the user and manages thegeneration of the Augmented Content 190. The Managed RAC 145 subsystemgenerates three outputs taking into consideration a user's feedback orinput. The Managed RAC 145 subsystem generates and controls thecommunication of the generated Augmented Content 190 using Display Queue165 and Display RAC 170. In addition, Managed RAC 145 generates anupdate request to Update RAC 155 for any necessary update to the DisplayQueue 165 based on a user's interaction or input. The Display Queue 165displays in a desired skin at least a portion of the queue of RACs sothat the user can browse through them and select some to view. TheDisplay Queue 165 displays a link, a summary, or a portion of thecompiled relevant content or pages. Upon selection or interaction by auser with one of the displayed RACs, the Display RAC 170 retrieves therespective relevant page RAC and displays at least a portion of it. TheDisplay RAC 170 subsystem manages and controls the displaying of theAugmented Content 190 using the display screen. Display RAC 170 can useone or more display layers on top of the Reference Content 105 or LocalContent 110 via translucent display layers as discussed in previousparagraphs.

A block diagram of a Hierarchical Augmentation System 200 is shown inFIG. 2. This hierarchical augmentation or nested augmentation capabilityenables a user to augment any content that is the result of dataaugmentation at any level of browsing or exploration. For example, giventhat the Augmentation System 210 generates a list of RACs, the user mayselect any one of the RACs or a group of RACs to invoke the augmentationsystem on and to generate another level of augmentation. TheAugmentation System 200 allows the user to go back and forth in thehierarchical graph to browse any particular content at any level, be ita reference or augmented content. For example, the Augmentation System200 provides augmented content at Process 1 220, which is the firstinvocation of the augmentation system on reference content, a user mayelect to augment one or more of the augmented content of Process 1 220.The Augmentation System 200 uses the elected content to be augmentedfrom Process 1 220 as an input or reference content to Process 2 230 foraugmentation. Process 2 230, which is considered the second invocationof the augmentation system on a reference content, generates in turnaugmented content which the user can further refine or interact with,and so on for Process k 340, Process (n−1) 350, and Process (n) 360.Multilevel nesting or hierarchical augmentation is not limited to aspecific number of levels. Of course certain hardware or softwarelimitations or a particular application may dictate the use of aspecific number of levels. However, this is an option that can be usedto various extents as part of the customization of Augmentation System200 for any particular usage.

A block diagram of a Hierarchical Augmentation System 300 is shown inFIG. 3. This hierarchical augmentation or nested augmentation capabilitycomprises the same capabilities as the Hierarchical Augmentation System200 is shown in FIG. 2 and includes an Augmentation System Control 390subsystem that is communicating Augmented and Reference contents AR-325,AR-335, AR-345, AR355, and AR-365 with Process 1 320, Process 2 330,Process k 340, Process (n−1) 350, and Process (n) 360, respectively. Asdescribed above Process 1 320 corresponds to a first level instance ofAugmented System 100, and Process (n) corresponds to an n-th levelinstance of Augmented System 100. Given that each of the nestedaugmented systems may generate different augmentation content for eachhierarchical level at least due to variations in user input or thereference content corresponding to the hierarchical level, theAugmentation System Control 390 may receive one or more of the generatedaugmented content of each hierarchical level, a copy of the set offilters, a copy of the set of features, and a copy of the set ofcategories. The Augmentation System Control 390 can further runsophisticated statistics, analytics and algorithms to extract newfeatures or generate new filters or categories. Furthermore, theAugmentation System Control 390 may receive user input to control whattype of analysis or augmentation the user expects the HierarchicalAugmentation System 300 to provide or keep track of nested contents thatthe user is interacting with, viewing, or manipulating at various levelsof hierarchy.

A block diagram of an Augmentation System 400 using a Display Control420 subsystem is shown in FIG. 4. The Augmentation System 410 isessentially the same as any one of the Augmentation System 100,Augmentation System 200 and Augmentation System 300 as shown in FIG. 1,FIG. 2, and FIG. 3 respectively. The Display Control 420 subsystemcontrols the displaying of various elements such as Augmented Content450 and Reference Content 440, which are output of the AugmentationSystem 410. In addition, the Display Control 420 receives input controlfrom Augmentation Display 430 subsystem and/or from a user interactingwith the Augmentation Display 430 or one or more display layersdisplayed using the Augmentation Display 430. Based on the AugmentedContent 450 generated from Augmentation System 410, Display Control 420generates and/or controls different display layers, widgets, icons, andother knobs which are utilized to show, control, or manipulate any oneof the Augmented Content 450 and Reference Content 440. Furthermore,Display Control 420 provides means for the user to interact with any oneof the Reference Content 440 or the Augmented Content 450.

In accordance with one embodiment, the Augmentation System 410, theDisplay Control 420, and Augmentation Display 430 are elements of thesame physical electronic system such as a mobile device. The user canmanipulate any one of the Augmented Content 450, Reference Content 440,and how each is displayed onto the Augmentation Display 430. Inaddition, a user interface (UI) may be used to further aid the user tomanipulate or interact with any one of the Reference Content 440 and theAugmented Content 450 and the displaying of such content. Furthermore,the UI can provide an easy mechanism for a user to interact with thecategories, widgets, buttons, and any other option that is presented forthe user to engage with the Augmentation System 410.

In accordance with one embodiment, the Augmentation System 410, and theDisplay Control 420 are elements of a first electronic device that isseparate from a second electronic device comprising the AugmentationDisplay 430, wherein the first and second electronic devices communicatethe Reference Content 440 and the Augmented Content 450 back and forthbased on the Augmentation System 410 and/or a user interaction with anyone of Reference Content 440 and Augmented Content 450.

In accordance with one embodiment, the Augmentation Display 430, and theDisplay Control 420 are elements of a first electronic device that isseparate from a second electronic device comprising the AugmentationSystem 410, wherein the first and second electronic devices communicatethe Reference Content 440 and the Augmented Content 450 back and forthbased on the Augmentation System 410 and/or a user interaction with anyone of Reference Content 440 and Augmented Content 450.

In accordance with one embodiment, a system for extraction andgeneration of relevant augmented content extraction is shown in FIG. 5.The Relevant Augmented Content Extraction 500 is tasked with building aset of features and categories that any one of the Augmentation System100, Augmentation System 200, Augmentation System 300 and AugmentationSystem 400 can utilize to generate augmented content. Reference Content510 is similar to Reference Content 105, and Local Content 520 issimilar to Local Content 110. Categories 530 is a subsystem which isresponsible for constructing a list of categories that captures or isresponsive to the user's inputs and preferences, a set of extractedcategories from Reference Content 510 and Local Content 520, and a setof customized categories associated with the user. Features and Metrics525 is a subsystem which generates a set of features, a set ofsignatures, and/or a set of metrics each of which is either dynamicallygenerated or pre-computed and stored. Features and Metrics 525 deliversthese sets of features to an Extract Features 540 subsystem. Inaddition, the Extract Features 540 receives input from Reference Content510, Local Content 520, Features and Metrics 525, and Categories 530.Extract Features 540 delivers a set of features, a set of signatures,and a set of metrics to Compile RAC 550 subsystem, which in turnutilizes one or more of those sets to compile from the internet, a localdata store, or any other data repository (public or private) a set ofdata elements corresponding to the relevant augmented content to bepresented to a user. A Relevant Augmented Content 560 subsystem receivesthe set of data elements and prioritizes the relevant augmented contentin accordance with a user preference or interaction, the set of featuresor filters, the set of signatures, and the set of metrics to generate acustomized augmented content for the user.

In accordance with one embodiment, a simplified block diagram of asystem for a relevant augmented content extraction is shown in FIG. 6.The Relevant Augmented Content Extraction 600 can be used as a part ofan augmentation system such as Augmentation System 100, AugmentationSystem 200, Augmentation System 300 and Augmentation System 400 each ofwhich has been described above. The Extract Relevant Features 600 isutilized to compile a set of features, using Compile RAC 650, to be usedby an augmentation system to generate augmented content. ExtractCandidate 618 processes at least a portion of a Reference Content 608,and receives other user-provided input to extract or generate one ormore set of filters and features. Features 620 uses the one or more setof filters and features to organize, build, compile or store auser-customized network of features, concepts, objects and theirrelationships. Features 620 serve to provide a better extraction or afocused extraction of a user's relevant set of features that can providea faster convergence on what the user is interested to see or would wantto see regarding the Reference Content 608. In addition, this provides abetter value add augmented content for the purpose of knowledgediscovery, learning, and a richer user experience in browsing and/orinteracting with data information. Furthermore, Features 620 can learn,save and further refine the user-customized network of features,concepts, objects and their relationships over time for richer and moreefficient user experience. Similarly, Categories 630 uses the one ormore set of filters and features generated by Extract Candidate 618 toorganize, build, compile or store a user-customized network ofcategories and their relationships. Categories 630 can learn, save andfurther refine the user-customized network of categories and theirrelationships over time for richer and more efficient user experience.

Metrics 640 is a system that can provide user influenced metricsinformation to Compile RAC 650. Metrics 640 uses the one or more set offilters and features generated by Extract Candidate 618 to organize,build, compile or store a user-customized network of metrics which canbe user defined or system's default. For example, Metrics 640 can usedate or time as a metric that can be used to further narrow and focus onthe relevance of the augmented content to the user or to the ReferenceContent 608. Another example is to use a source or a group of sources toaid Compile RAC 650 to limit or expand its compilation and generation ofrelevant augmented content. Metrics 640 can learn, save and furtherrefine the user-customized network of metrics and their relationshipsover time for richer and more efficient user experience. Metrics 640 canreceive real time information from the user or other part of anaugmentation system, and provides an update in real time to Compile RAC650.

Compile RAC 650 is used to compile the networks of features, categoriesand metrics received from Features 620, Categories 630, and Metrics 640to generate and prioritize a focused set of relevant augmented content(RAC) that captures the properties and/or attributes of ReferenceContent 608 and reflects the user's rules, interests, preferences, andattributes. This focused set of relevant augmented content (RAC) is tobe used by an augmentation system to deliver or present a concise andhighly relevant augmented content to the user. Compile RAC 650 is usedto resolve any conflicts that may exist between any of the networks offeatures, categories and metrics. Compile RAC 650 also provides anddetermines the priority of the final list of RACs to be delivered orpresented to the user. Compile RAC 650 can also receive, generate ormodify an association rule which can be used to leverage, or joins oneor more categories, features, filters, concepts, or metrics to (i)generate a new set of features, filters, categories, or relevantaugmented content, and (ii) modify one or more of the set of features,filters, or categories which are being used to generate the relevantaugmented content.

In accordance with one embodiment, an augmentation system can useDisplay Layers and Controls 700 as shown in FIG. 7 to display thegenerated Augmented Content 760, Global Augmented Content Queue 720,Global Augmented Content Queue 770, and the Reference Content 750. Forexample, this can be one instantiation of the data presentationmechanism of an augmentation system as described above, e.g AugmentationSystem 100. The user can change the look and feel (skin) of the DisplayLayers and Controls 700 using any number of skins (look and feeloptions). The Global Augmented Content Queue 720 corresponds to adisplayed part of a relevant augmented content (RAC) generated by theaugmentation system. The user can browse and scroll through this queueto select a relevant augmented content of interest. Local AugmentedContent Queue 770 refers to the relevant augmentation results that arerelated to the part of Reference Content 750 that the user hasinteracted with or is being displayed via Display Screen 710, and whichis referred to as Local Content. Display Layers and Controls 700 canmanage the display of the Global Augmented Content Queue 720 and LocalAugmented Content Queue 770 in various ways, such as the location of thedisplay of the queues as well as the portion of any one of the queuesthat is being displayed using Display Screen 710. For example, the usercan choose that only the Global Augmented Content Queue 720 isdisplayed, thus Display Layers and Controls 700 will manage to displaythe portion of RACs of the Global Augmented Content Queue 720 that maybeaccommodated onto the Display Screen 710. Similarly, the user may chooseto emphasize the Local Augmented Content Queue 720 and thus the DisplayLayers and Controls 700 will manage that as well. The Reference Content750 refers to the content being browsed and explored for furtheraugmentation. Display Screen 710 corresponds to a display screen thatmay be physically collocated within the same device where theaugmentation system is being used, or it can be part of a separateelectronic device. Augmented Content 760 is displayed using one or moredisplay layers, and is the augmentation content that the user chooses toview. An icon Promote 740 is used to highlight, select, or promote aspecific RAC. Promote 740 provides a mechanism for the user to interactwith any of the RACs of Global Augmented Content Queue 720 and LocalAugmented Content 770 by elevating the priorities of a RACs. Similarly,a demote icon (not shown) can be used by the user to remove or dismiss aRAC or a group of RACs entirely if the user is not interested in them.

A Simulated Display 800 is a use case scenario of the Display Layers andControls 700 and any one of the augmentation systems described earlieras shown in FIG. 8. This Simulated Display 800 presents an example of auser reading an article about AIDS as shown in Reference Content 810.The user then invokes the Augmentation System to augment ReferenceContent 810. Based on the categories presented to the user, the userselects categories that are related to AIDS Research and Science.Categories Related to AIDS RACs 820 shows part of the globalaugmentation deep queue that the system generated in response to theuser's interest in AIDS, Science, and Research.

A Simulated Display 900 is a use case scenario of the Display Layers andControls 700 and any one of the augmentation systems described earlieras shown in FIG. 9. This Simulated Display 900 presents an example of auser reading an article about AIDS as shown in Reference Content 910.Selected Content 920 shows an example of selecting part of the ReferenceContent 910. The user then invokes the Augmentation System to augmentReference Content 910 and Selected Content 920. Based on the user'schoice of categories related to Science and Research of AIDS, and thesystem's extracted categories and features RACs 905 shows part of theglobal augmentation deep queue that the system generated. Africa/IndiaAIDS RACs 930 shows part of the local augmentation deep queue that theAugmentation System generates in response to the user's selection ofpart of Reference Content 910. Augmented Display Layer 940 is an exampleof displaying of Africa/India AIDS RACs 930. Augmented Display Layer 940shows a RAC (HIV and AIDS) in the local augmentation queue that theelected to view.

While embodiments, implementations, and applications have been shown anddescribed, it would be apparent to those skilled in the art having thebenefit of this disclosure that many more modifications than mentionedabove are possible without departing from the inventive conceptsdisclosed herein.

I claim:
 1. A system for generating and presenting augmented content,the system comprising: an electronic device comprising an electronicdisplay, the electronic display configured to display at least a portionof a reference content using a first display layer, the first displaylayer is displayed using at least a portion of the electronic display;and an augmentation system configured to generate: a first set offilters using a first collection of relevant features associated with atleast one of the reference content and the at least a portion of thereference content, a second set of filters using a second collection ofrelevant features associated with a user, a first set of categoriesusing the first set of filters and the second set of filters; a firstset of augmented content, the first set of augmented content includesone or more groups of augmented content for each category of the firstset of categories, each group of the one or more groups of augmentedcontent includes at least one augmented content, wherein each group ofthe one or more groups of augmented content is generated using at leastone category of the first set of categories, wherein at least a portionof the first set of augmented content is displayed using a seconddisplay layer, and wherein the user is able to interact with any one ofthe displayed portion of the reference content and the displayed portionof the first set of augmented content using the electronic display. 2.The system of claim 1, wherein the second display layer is displayed ontop of the first display layer using the electronic display.
 3. Thesystem of claim 2, wherein the electronic display controls the seconddisplay layer translucency using a programmable value.
 4. The system ofclaim 2, wherein the user interaction with any one of the displayedportion of the reference content and the displayed portion of the firstset of augmented content comprises at least one of a manipulation of atleast a portion of the reference content displayed using any one of aregion of the first display layer, a manipulation of at least a portionof the first set of augmented content displayed using a region of thesecond display layer, hiding of the first display layer, hiding of thesecond display layer, modifying translucency of the second displaylayer, a selection of a region of the first display layer, and amanipulation of a region of the first display layer.
 5. The system ofclaim 1 wherein at least a part of the displayed portion of thereference content is visible through at least a part of the seconddisplay layer.
 6. The system of claim 1, further comprising: a displaycontrol configured to display at least a portion of one or more of thefirst set of filters, the second set of filters, the first set ofcategories, a set of metrics, and a set of user preferences.
 7. Thesystem of claim 6, wherein the augmentation system configured to save atleast a portion of one or more of the first set of filters, the secondset of filters, the first set of categories, a set of metrics, and a setof user preferences using a memory subsystem of the electronic device ora memory subsystem of the electronic display.
 8. The system of claim 7,wherein the augmentation system configured to generate a new set ofaugmented content using one or more of the first set of filters, thesecond set of filters, the first set of categories, the set of metrics,and the set of user preferences.
 9. The system of claim 1, wherein anyone of the reference content and the at least one augmented contentincludes at least one of a user-controlled content, a content from anonline data provider, a content from an online service provider, acontent from an online search engine, a content from an enterpriseserver, a content from a database, a content from a web page, a contentusing at least a portion of a previously generated augmented content,and a linked content.
 10. The system of claim 1, wherein the firstdisplay layer having a first size and at least one pixel located at afirst position within the electronic display, the second display layerhaving a second size and at least one pixel located at a second positionwithin the electronic display, and wherein the user is able tomanipulate at least one of the first size, the first position, thesecond size, and the second position based on at least one of aselection of an icon, a manipulation of a region of the first displayscreen, an activation of a button, and one or more user gesture on theelectronic display.
 11. The system of claim 1, wherein the electronicdevice communicates with the electronic display using at least one of awired communication and a wireless communication.
 12. The system ofclaim 1, wherein the electronic device includes any one or more of amobile device, a handheld device, a human interface device, anelectronic display device, a desktop monitor, a desktop computer, apersonal computing device, a remote server, a touch-based input device,and a tablet based device.